Head mounted display and display for selectively displaying a synthesized image and a physical space image, and control method thereof

ABSTRACT

A display control unit causes a display device to display the synthesized image in a case where the determination unit does not determine that the error region exists in the synthesized image, and causes the display device to display the physical space image in a case where the determination unit does determine that the error region exists in the synthesized image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of application Ser. No. 12/127,583,filed May 27, 2008, the entire disclosure of which is herebyincorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a technique of displaying an image.

2. Description of the Related Art

Mixed reality, that is, so-called MR is recently known as a technologyof merging a real world and a virtual world seamlessly in real time. Inone MR technology, a video see-through HMD (Head Mounted Display) isused and an object almost matching an object observed from the pupilposition of an HMD wearer is sensed by, for example, a video camera. ACG (Computer Graphics) image is superimposed on the sensed image togenerate an image. The generated image is presented in front of the eyesof the HMD wearer.

FIG. 12 is a block diagram showing the functional arrangement of ageneral video see-through mixed reality system (to be referred to as asystem hereinafter). A general system will be described below withreference to FIG. 12.

As shown in FIG. 12, the system comprises an image processing apparatus1202 and a video see-through HMD 1201.

The video see-through HMD 1201 includes an image sensing unit 1203, adisplay unit 1204, a three-dimensional (3D) position sensor 1205, and aI/F (interface) 1206. The image processing apparatus 1202 is generallyan apparatus such as a personal computer or workstation with an advancedcalculation processing function and graphic display function, andincludes an I/F 1207, position and orientation information generationunit 1208, and CG rendering composition unit 1210.

The components forming the video see-through HMD 1201 will be describedfirst.

The image sensing unit 1203 senses an image (physical space image) ofthe outside world which can be seen, in almost the same direction as adirection of line of sight, from a position almost matching theviewpoint of the wearer who is wearing the video see-through HMD 1201 onthe head. The image sensing unit 1203 is generally provided for each ofthe right and left eyes to generate a stereoscopic image, and includesan image sensing element, an optical system, and a DSP (Digital SignalProcessor) that executes an image process.

The display unit 1204 displays an MR image output from the imageprocessing apparatus 1202. The display unit 1204 is also provided foreach of the right and left eyes, and includes a display device andoptical system. A small liquid crystal display device or a retina scantype device by MEMS (Micro Electro Mechanical System) is used as thedisplay device.

The 3D position sensor 1205 measures the position and orientation ofitself. A magnetic sensor or a gyro sensor (acceleration and angularvelocity) is used as the 3D position sensor 1205.

The I/F 1206 interconnects the video see-through HMD 1201 and the imageprocessing apparatus 1202. The image sensed by the image sensing unit1203 and a measurement result by the 3D position sensor 1205 aretransferred to the image processing apparatus 1202 via the I/F 1206. TheMR image generated on the image processing apparatus 1202 side is inputto the video see-through HMD 1201 via the I/F 1206. The I/F 1206 that isrequired to transmit an enormous quantity of data in real time uses aconnector compatible with a metal line such as a USB or IEEE1394 or anoptical fiber such as GigabitEthernet®.

The components of the image processing apparatus 1202 will be describednext.

The I/F 1207 interconnects the image processing apparatus 1202 and thevideo see-through HMD 1201. The image sensed by the image sensing unit1203 and the measurement result by the 3D position sensor 1205 aretransferred to the image processing apparatus 1202 via the I/F 1207. TheMR image generated on the image processing apparatus 1202 side is inputto the video see-through HMD 1201 via the I/F 1207.

The position and orientation information generation unit 1208 calculatesposition and orientation information representing the position andorientation of the viewpoint of the video see-through HMD 1201 wearerbased on the measurement result by the 3D position sensor 1205 which hasbeen received from the video see-through HMD 1201. Alternatively, amethod of generating the position and orientation information of theviewpoint by using a marker or the like within the image sensed by theimage sensing unit 1203 may also be used, as a matter of course.

Contents 1209 form a storage device which stores data pertaining tovirtual objects forming the virtual space.

The CG rendering composition unit 1210 generates an image (CG image), ofthe virtual space according to the data of the contents 1209, which isseen from the viewpoint having the position and orientation representedby the position and orientation information generated by the positionand orientation information generation unit 1208. The CG renderingcomposition unit 1210 composites the CG image on the image received fromthe video see-through HMD 1201 via the I/F 1207, and generates an MRimage. The CG rendering composition unit 1210 outputs the generated MRimage to the video see-through HMD 1201 via the I/F 1207.

With the above-described arrangement, the video see-through HMD 1201wearer can experience a mixed reality world obtained by merging a realworld and a virtual world seamlessly in real time.

In the above-mentioned system using the video see-through HMD, it isimportant to secure sight of the wearer, even if a communication erroroccurs between the HMD and the image processing apparatus. Inparticular, when a wireless scheme is employed for image transmission,errors may occur frequently depending on the environment of use and thedistance between the apparatuses. In one technique, when suchcommunication error occurs, a frame of a predetermined time before acarrier wave disappears in infrared ray video transmission is displayedas a still image together with a warning message (patent reference 1).

The following technique has also been proposed. That is, upon occurrenceof a decoding error in one channel, a stereoscopic video display systemincluding a plurality of cameras controls to transfer some or all ofimage data stored in another channel to the channel in which the errorhas occurred. The system interpolates some or all of image data in whichthe error has occurred, and displays the interpolated image (patentreference 2).

-   [Patent Reference 1] Japanese Patent Laid-Open No. 5-76078-   [Patent Reference 2] Japanese Patent Laid-Open No. 7-322302

However, the above-described prior art has the following problems.

Assume that an error occurs in wireless image transmission and it isdifficult to transmit the image. If an image of the successfullyreceived, immediately preceding frame is displayed, a wearer does notknow the actual outside world when moving. That is, it is difficult tosecure the user's sight.

Furthermore, when an interpolation process is executed as in anotherconventional example, as the amount and frequency of error increase, itbecomes difficult to reconstruct an image even if an identical image,stereoscopic image, or a preceding frame is used. Therefore, it is alsodifficult to secure the wearer's sight.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the aboveproblems, and has as its object to provide a technique for, even if anerror portion appears within an image of a certain frame, providing asuitable image by correcting the error portion in a simpler and easierway, when the image is presented to both eyes of an observer.

According to the first aspect of the present invention, there isprovided a head mounted display having a display unit for displaying animage in front of eyes of an observer, comprising: an input unit adaptedto sequentially input images of frames; a determination unit adapted todetermine whether an error portion exists within an image of a frame ofinterest input by the input unit; and a display control unit adapted tocause the display unit to display the image of the frame of interestwhen the determination unit determines that no error portion existswithin the image of the frame of interest,

when the determination unit determines that the error portion existswithin the image of the frame of interest, the display control unitincluding a unit adapted to determine whether a correction process forthe image of the frame of interest is to be executed, on the basis ofthe size of the error portion within the image of the frame of interest,which has been calculated during the determination process by thedetermination unit, and a determination result determined by thedetermination unit for a frame before the frame of interest, and acontrol unit adapted to select, based on the size and the determinationresult, either display control to cause the display unit to display theimage of the frame before the frame of interest as the image of theframe of interest or display control to cause the display unit todisplay a result obtained by correcting the error portion within theimage of the frame of interest using a pixel group surrounding the errorportion, when it is determined that the correction process for the imageof the frame of interest is to be executed, and to execute the selecteddisplay control.

According to the second aspect of the present invention, there isprovided a head mounted display for displaying an image in front of eyesof an observer, comprising: a presentation unit for presenting an imageto one eye of the observer; and a presentation unit for presenting animage to the other eye, each of the presentation units including aninput unit adapted to sequentially input images of frames, adetermination unit adapted to determine whether an error portion existswithin an image of a frame of interest input by the input unit, and adisplay control unit adapted to cause a display unit to display theimage of the frame of interest when the determination unit determinesthat no error portion exists within the image of the frame of interest,and when the determination unit determines that the error portion existswithin the image of the frame of interest, the display control unitincluding a unit adapted to determine whether a correction process forthe image of the frame of interest is to be executed, on the basis ofthe size of the error portion within the image of the frame of interest,which has been calculated during the determination process by thedetermination unit, and a determination result determined by thedetermination unit for a frame before the frame of interest, and acontrol unit adapted to select, based on the size and the determinationresult, either display control to cause the display unit to display, asthe image of the frame of interest, the image of the frame of interestinput by the input unit of the other presentation unit or displaycontrol to cause the display unit to display a result obtained bycorrecting the error portion within the image of the frame of interestusing a pixel group surrounding the error portion, when it is determinedthat the correction process for the image of the frame of interest is tobe executed, and to execute the selected display control.

According to the third aspect of the present invention, there isprovided a head mounted display for displaying an image in front of eyesof an observer, comprising: a unit adapted Lo acquire a physical spaceimage obtained by sensing a physical space by an image sensing device; aunit adapted to acquire a position and orientation of the image sensingdevice; a unit adapted to acquire a virtual space image generated basedon the position and orientation; a unit adapted to detect thepresence/absence of an error for the virtual space image; and a displaycontrol unit adapted to display a composite image of the physical spaceimage and the virtual space image when the error is not detected, and todisplay the physical space image when the error is detected.

According to the fourth aspect of the present invention, there isprovided a head mounted display for displaying an image in front of eyesof an observer, comprising: a unit adapted to acquire a physical spaceimage obtained by sensing a physical space by an image sensing device; aunit adapted to acquire a position and orientation of the image sensingdevice; a unit adapted to acquire a composite image of the physicalspace image and a virtual space image generated based on the positionand orientation; a unit adapted to detect the presence/absence of anerror for the composite image; and a display control unit adapted todisplay the composite image when the error is not detected, and todisplay the physical space image when the error is detected.

According to the fifth aspect of the present invention, there isprovided a control method for a head mounted display having a displayunit which displays an image in front of eyes of an observer,comprising: an input step of sequentially inputting images of frames; adetermination step of determining whether an error portion exists withinan image of a frame of interest input in the input step; and a displaycontrol step of causing the display unit to display the image of theframe of interest when it is determined in the determination step thatno error portion exists within the image of the frame of interest, whenit is determined in the determination step that the error portion existswithin the image of the frame of interest, the display control stepincluding, a step of determining whether a correction process for theimage of the frame of interest is to be executed, on the basis of thesize of the error portion within the image of the frame of interest,which has been calculated during the determination process in thedetermination step, and a determination result determined for a framebefore the frame of interest in the determination step, and a controlstep of selecting, based on the size and the determination result,either display control to cause the display unit to display the image ofthe frame before the frame of interest as the image of the frame ofinterest or display control to cause the display unit to display aresult obtained by correcting the error portion within the image of theframe of interest using a pixel group surrounding the error portion,when it is determined that the correction process for the image of theframe of interest is to be executed, and of executing the selecteddisplay control.

According to the sixth aspect of the present invention, there isprovided a control method for a head mounted display which displays animage in front of eyes of an observer, wherein the head mounted displayincludes a presentation unit for presenting an image to one eye of theobserver and that for presenting an image to the other eye, each of thepresentation units executing an input step of sequentially input imagesof frames, a determination step of determining whether an error portionexists within an image of a frame of interest input in the input step,and a display control step of causing a display unit to display theimage of the frame of interest when it is determined in thedetermination step that no error portion exists within the image of theframe of interest, and when it is determined in the determination stepthat the error portion exists within the image of the frame of interest,the display control step including a step of determining whether acorrection process for the image of the frame of interest is to beexecuted, on the basis of the size of the error portion within the imageof the frame of interest, which has been calculated during thedetermination process in the determination step, and a determinationresult determined for a frame before the frame of interest in thedetermination step, and a control step of selecting, based on the sizeand the determination result, either display control to cause thedisplay unit to display, as the image of the frame of interest, theimage of the frame of interest input in the input step of the otherpresentation unit or display control to cause the display unit todisplay a result obtained by correcting the error portion within theimage of the frame of interest using a pixel group surrounding the errorportion, when it is determined that the correction process for the imageof the frame of interest is to be executed, and of executing theselected display control.

According to the seventh aspect of the present invention, there isprovided a control method for a head mounted display which displays animage in front of eyes of an observer, comprising: a step of acquiring aphysical space image obtained by sensing a physical space by an imagesensing device; a step of acquiring a position and orientation of theimage sensing device; a step of acquiring a virtual space imagegenerated based on the position and orientation; a step of detecting thepresence/absence of an error for the virtual space image; and a displaycontrol step of displaying a composite image of the physical space imageand the virtual space image when the error is not detected, anddisplaying the physical space image when the error is detected.

According to the eighth aspect of the present invention, there isprovided a control method for a head mounted display which displays animage in front of eyes of an observer, comprising: a step of acquiring aphysical space image obtained by sensing a physical space by an imagesensing device; a step of acquiring a position and orientation of theimage sensing device; a step of acquiring a composite image of thephysical space image and a virtual space image generated based on theposition and orientation; a step of detecting the presence/absence of anerror for the composite image; and a display control step of displayingthe composite image when the error is not detected, and displaying thephysical space image when the error is detected.

According to the ninth aspect of the present invention, there isprovided a display having a display unit for displaying an image infront of eyes of an observer, comprising: an input unit adapted tosequentially input images of frames; a determination unit adapted todetermine whether an error portion exists within an image of a frame ofinterest input by the input unit; and a display control unit adapted tocause the display unit to display the image of the frame of interestwhen the determination unit determines that no error portion existswithin the image of the frame of interest, when the determination unitdetermines that the error portion exists within the image of the frameof interest, the display control unit including a unit adapted todetermine whether a correction process for the image of the frame ofinterest is to be executed, on the basis of the size of the errorportion within the image of the frame of interest, which has beencalculated during the determination process by the determination unit,and a determination result determined by the determination unit for aframe before the frame of interest, and a control unit adapted toselect, based on the size and the determination result, either displaycontrol to cause the display unit to display the image of the framebefore the frame of interest as the image of the frame of interest ordisplay control to cause the display unit to display a result obtainedby correcting the error portion within the image of the frame ofinterest using a pixel group surrounding the error portion, when it isdetermined that the correction process for the image of the frame ofinterest is to be executed, and to execute the selected display control.

According to the tenth aspect of the present invention, there isprovided a display for displaying an image in front of eyes of anobserver, comprising: a presentation unit for presenting an image to oneeye of the observer; and a presentation unit for presenting an image tothe other eye, each of the presentation units including an input unitadapted to sequentially input images of frames, a determination unitadapted to determine whether an error portion exists within an image ofa frame of interest input by the input unit, and a display control unitadapted to cause a display unit to display the image of the frame ofinterest when the determination unit determines that no error portionexists within the image of the frame of interest, and when thedetermination unit determines that the error portion exists within theimage of the frame of interest, the display control unit including aunit adapted to determine whether a correction process for the image ofthe frame of interest is to be executed, on the basis of the size of theerror portion within the image of the frame of interest, which has beencalculated during the determination process by the determination unit,and a determination result determined by the determination unit for aframe before the frame of interest, and

a control unit adapted to select, based on the size and thedetermination result, either display control to cause the display unitto display, as the image of the frame of interest, the image of theframe of interest input by the input unit of the other presentation unitor display control to cause the display unit to display a resultobtained by correcting the error portion within the image of the frameof interest using a pixel group surrounding the error portion, when itis determined that the correction process for the image of the frame ofinterest is to be executed, and to execute the selected display control.

According to the eleventh aspect of the present invention, there isprovided a display for displaying an image in front of eyes of anobserver, comprising: a unit adapted to acquire a physical space imageobtained by sensing a physical space by an image sensing device; a unitadapted to acquire a position and orientation of the image sensingdevice; a unit adapted to acquire a virtual space image generated basedon the position and orientation; a unit adapted to detect thepresence/absence of an error for the virtual space image; and a displaycontrol unit adapted to display a composite image of the physical spaceimage and the virtual space image when the error is not detected, and todisplay the physical space image when the error is detected.

According to the twelfth aspect of the present invention, there isprovided a display for displaying an image in front of eyes of anobserver, comprising: a unit adapted to acquire a physical space imageobtained by sensing a physical space by an image sensing device; a unitadapted to acquire a position and orientation of the image sensingdevice; a unit adapted to acquire a composite image of the physicalspace image and a virtual space image generated based on the positionand orientation; a unit adapted to detect the presence/absence of anerror for the composite image; and a display control unit adapted todisplay the composite image when the error is not detected, and todisplay the physical space image when the error is detected.

According to the thirteenth aspect of the present invention, there isprovided a control method for a display having a display unit whichdisplays an image in front of eyes of an observer, comprising: an inputstep of sequentially inputting images of frames; a determination step ofdetermining whether an error portion exists within an image of a frameof interest input in the input step; and a display control step ofcausing the display unit to display the image of the frame of interestwhen it is determined in the determination step that no error portionexists within the image of the frame of interest, when it is determinedin the determination step that the error portion exists within the imageof the frame of interest, the display control step including a step ofdetermining whether a correction process for the image of the frame ofinterest is to be executed, on the basis of the size of the errorportion within the image of the frame of interest, which has beencalculated during the determination process in the determination step,and a determination result determined for a frame before the frame ofinterest in the determination step, and a control step of selecting,based on the size and the determination result, either display controlto cause the display unit to display the image of the frame before theframe of interest as the image of the frame of interest or displaycontrol to cause the display unit to display a result obtained bycorrecting the error portion within the image of the frame of interestusing a pixel group surrounding the error portion, when it is determinedthat the correction process for the image of the frame of interest is tobe executed, and of executing the selected display control.

According to the fourteenth aspect of the present invention, there isprovided a control method for a display which displays an image in frontof eyes of an observer, wherein the display includes a presentation unitfor presenting an image to one eye of the observer and that forpresenting an image to the other eye, each of the presentation unitsexecuting an input step of sequentially input images of frames, adetermination step of determining whether an error portion exists withinan image of a frame of interest input in the input step, and a displaycontrol step of causing a display unit to display the image of the frameof interest when it is determined in the determination step that noerror portion exists within the image of the frame of interest, and whenit is determined in the determination step that the error portion existswithin the image of the frame of interest, the display control stepincluding a step of determining whether a correction process for theimage of the frame of interest is to be executed, on the basis of thesize of the error portion within the image of the frame of interest,which has been calculated during the determination process in thedetermination step, and a determination result determined for a framebefore the frame of interest in the determination step, and a controlstep of selecting, based on the size and the determination result,either display control to cause the display unit to display, as theimage of the frame of interest, the image of the frame of interest inputin the input step of the other presentation unit or display control tocause the display unit to display a result obtained by correcting theerror portion within the image of the frame of interest using a pixelgroup surrounding the error portion, when it is determined that thecorrection process for the image of the frame of interest is to beexecuted, and of executing the selected display control.

According to the fifteenth aspect of the present invention, there isprovided a control method for a display which displays an image in frontof eyes of an observer, comprising: a step of acquiring a physical spaceimage obtained by sensing a physical space by an image sensing device; astep of acquiring a position and orientation of the image sensingdevice; a step of acquiring a virtual space image generated based on theposition and orientation; a step of detecting the presence/absence of anerror for the virtual space image; and a display control step ofdisplaying a composite image of the physical space image and the virtualspace image when the error is not detected, and displaying the physicalspace image when the error is detected.

According to the sixteenth aspect of the present invention, there isprovided a control method for a display which displays an image in frontof eyes of an observer, comprising: a step of acquiring a physical spaceimage obtained by sensing a physical space by an image sensing device; astep of acquiring a position and orientation of the image sensingdevice; a step of acquiring a composite image of the physical spaceimage and a virtual space image generated based on the position andorientation; a step of detecting the presence/absence of an error forthe composite image; and a display control step of displaying thecomposite image when the error is not detected, and displaying thephysical space image when the error is detected.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the functional arrangement of a systemaccording to the first embodiment of the present invention;

FIG. 2 is a flowchart of a process executed by a video see-through HMD101;

FIG. 3 is a view showing an example of a physical space sensed by animage sensing unit 103;

FIG. 4 is a view showing an example of an image transmitted/receivedbetween the video see-through HMD 101 and an image processing apparatus102;

FIG. 5 is a view for explaining a process of substituting an MR image ofan immediately preceding frame for that of the frame of interest;

FIG. 6 is a view for explaining a process of correcting an error portionby using a pixel group surrounding it;

FIG. 7 is a block diagram showing a functional arrangement example of asystem according to the second embodiment of the present invention;

FIG. 8 is a flowchart of a process executed by a video see-through HMD701;

FIG. 9 is a view for explaining a process of substituting an MR imagefor one eye for an MR image for the other eye in order to correct anerror portion within the MR image for the other eye;

FIG. 10 is a block diagram showing a functional arrangement example of asystem according to the third embodiment of the present invention;

FIG. 11 is a flowchart of a process executed by a video see-through HMD1001;

FIG. 12 is a block diagram showing the functional arrangement of ageneral video see-through mixed reality system;

FIG. 13 is a block diagram showing a functional arrangement example of asystem according to the fourth embodiment of the present invention;

FIG. 14 is a block diagram showing a functional arrangement example ofan abnormality determination unit 1309;

FIG. 15 is a block diagram showing a functional arrangement example of asystem according to the fifth embodiment of the present invention; and

FIG. 16 is a block diagram showing a functional arrangement example ofan abnormality determination unit 1409.

DESCRIPTION OF THE EMBODIMENTS

Preferred embodiments of the present invention will be described indetail hereinafter with reference to the accompanying drawings. Notethat these embodiments will be described as examples of preferredarrangements of the invention described in the scope of claims, and suchinvention is not limited to the embodiments to be described hereinafter.

[First Embodiment]

FIG. 1 is a block diagram showing the functional arrangement of a systemaccording to this embodiment. As shown in FIG. 1, the system accordingto this embodiment comprises an image processing apparatus 102represented by a computer such as a PC (Personal Computer), and a videosee-through HMD 101 as an example of a head mounted display. The imageprocessing apparatus 102 and the video see-through HMD 101 wirelesslycommunicate with each other.

The video see-through HMD 101 will be described first.

An image sensing unit 103 is attached at a position near an eye (eitherthe left or right eye) of an observer wearing the video see-through HMD101 on the head so as to face in a direction of line of sight of theobserver. The image sensing unit 103 is a video camera (image sensingdevice) which senses a movie. Images (physical space images) of thesensed frames are sequentially output to a subsequent OSD (On ScreenDisplay) generation unit 114 and wireless I/F 104. The wireless I/F 104sequentially wirelessly transmits the images of the frames received fromthe image sensing unit 103 to the image processing apparatus 102.

Details of processing will be described later. In brief, the imageprocessing apparatus 102 transmits to the wireless I/F 104 an MR image(composite image) formed by superimposing a virtual space image on aphysical space image. The wireless I/F 104 receives (inputs) thetransmitted MR image, and outputs it to a subsequent error detectionunit 109 and interpolation unit 112.

Each time the composite image is received from the wireless I/F 104, theinterpolation unit 112 stores the received composite image in a framebuffer 113.

The error detection unit 109 checks (determines) whether an errorportion (error region) exists within the composite image received fromthe wireless I/F 104. In this check process, a general error check suchas a parity check, checksum, or CRC (Cyclic Redundancy Check) is used.The error detection unit 109 outputs the check result to a subsequenterror region/frequency determination unit 110.

The error region/frequency determination unit 110 refers to thedetermination result by the error detection unit 109. If the errordetection unit 109 detects no error portion, the error region/frequencydetermination unit 110 informs a selector 111 of it. If the errordetection unit 109 detects an error portion, by using “the size of theerror portion” acquired during the determination process and “a resultdetermined by the error detection unit 109 for an image of a pastframe”, the error region/frequency determination unit 110 executes adetermination process to be described later. This determination processis executed as follows. That is, it is determined whether the errorportion within a composite image of the current frame is to be corrected(interpolated) by substituting a composite image of the immediatelypreceding frame for that of the current image or by using a pixel groupsurrounding the error portion, or whether it is impossible to correctthe error portion. The error region/frequency determination unit 110informs the subsequent selector 111 of the determination result.

If the error region/frequency determination unit 110 informs theselector 111 that the error detection unit 109 detects no error portion,the selector 111 instructs the interpolation unit 112 to read out thecomposite image of the current frame stored in the frame buffer 113. Theinterpolation unit 112 reads out the composite image in accordance withthe instruction, and outputs the readout image to the selector 111. Theselector 111 outputs the composite image to a display unit 115.

When the error region/frequency determination unit 110 informs theselector 111 that the composite image of the immediately preceding frameis to be substituted for the composite image of the current frame inorder to correct the error portion, the selector 111 operates asfollows. The selector 111 instructs the interpolation unit 112 to readout the composite image of the frame immediately before the currentframe stored in the frame buffer 113. The interpolation unit 112 readsout the composite image in accordance with the instruction, and outputsthe readout image to the selector 111. The selector 111 outputs thecomposite image to the display unit 115.

When the error region/frequency determination unit 110 informs theselector 111 that the error portion is to be corrected by using thepixel group surrounding the error portion, the selector 111 operates asfollows. The selector 111 instructs the interpolation unit 112 to readout the composite image of the current frame stored in the frame buffer113. The interpolation unit 112 reads out the composite image inaccordance with the instruction, and corrects the error portion withinthe composite image in a process to be described later. Theinterpolation unit 112 outputs the corrected composite image to theselector 111. The selector 111 outputs the composite image to thedisplay unit 115.

When the error region/frequency determination unit 110 informs theselector 111 that it is impossible to correct the error portion, theselector 111 operates as follows. That is, the selector 111 instructsthe OSD generation unit 114 to superimpose message informationrepresenting a warning on the physical space image which has beenacquired by the OSD generation unit 114 from the image sensing unit 103as the physical space image of the current frame. The messageinformation is, for example, text data representing that the virtualspace image cannot be composited. The message information is not limitedto this, and may take various forms, as a matter of course. The selector111 outputs to the display unit 115 the physical space image on whichthe message information is superimposed.

Upon reception of the image from the selector 111, the display unit 115displays it. The display unit 115 is attached to the video see-throughHMD 101 so as to be located in front of the eyes of the observer wearingthe video see-through HMD 101 on the head. The image output from theselector 111 is therefore displayed in front of the observer's eyeswearing the video see-through HMD 101 on the head.

The image processing apparatus 102 will be described next.

Upon reception of the images of the frames transmitted from the wirelessI/F 104, a wireless I/F 105 sequentially outputs them to a subsequentposition and orientation information generation unit 106.

The position and orientation information generation unit 106 uses theimage received from the wireless I/F 105 to calculate the position andorientation information of the video see-through HMD 101. A process forcalculating the position and orientation information by using the imagewill be described later. The calculated position and orientationinformation is output to a subsequent CG rendering composition unit 107.

Contents 108 form a storage device which stores a data group pertainingto virtual objects forming the virtual space. For example, the datagroup includes initial position and orientation data of virtual objects.When each virtual object is formed by polygons, the data group includesdata of normal vectors and colors of the polygons, and coordinate valuedata of vertices which form each polygon.

The CG rendering composition unit 107 generates, as a virtual spaceimage, an image, of the virtual space according to the data of thecontents 108, which is seen from the viewpoint having the position andorientation represented by the position and orientation informationreceived from the position and orientation information generation unit106. The CG rendering composition unit 107 superimposes the generatedvirtual space image on the physical space image received from thewireless I/F 105, and generates a composite image as the MR image. TheCG rendering composition unit 107 outputs the generated MR image to thewireless I/F 105. The wireless I/F 105 wirelessly transmits the MR imageto the video see-through HMD 101.

FIG. 2 is a flowchart of a process executed by the video see-through HMD101. The process according to the flowchart shown in FIG. 2 is that ofcausing the display unit 115 to display an MR image of a frame, and inpractice the process is repeated. This enables the display unit 115 todisplay MR images of a plurality of frames. Note that a process ofcausing the display unit 115 to display an MR image of the frame ofinterest (each of the second and subsequent frames) will be describedbelow for descriptive convenience.

In step S201, the image sensing unit 103 acquires an image of the frameof interest by sensing a physical space seen from the position of theimage sensing unit 103 in a sensing direction. The image sensing unit103 outputs the acquired image to the wireless I/F 104 and OSDgeneration unit 114.

In step S202, the wireless I/F 104 wirelessly transmits to the imageprocessing apparatus 102 the image of the frame of interest which hasbeen acquired in step S201.

The image processing apparatus 102 executes the above-described processusing the image transmitted in step S202, generates an MR image of theframe of interest, and wirelessly transmits it to the video see-throughHMD 101 via the wireless I/F 105.

In step S207, the wireless I/F 104 receives the MR image of the frame ofinterest which has been transmitted from the image processing apparatus102. The wireless I/F 104 then outputs the received MR image to theerror detection unit 109 and interpolation unit 112. The interpolationunit 112 stores the MR image of the frame of interest in the framebuffer 113.

In step S208, the error detection unit 109 determines whether an errorportion exists within the MR image of the frame of interest. The errordetection unit 109 informs the error region/frequency determination unit110 of the determination result. If the determination result representsthat an error portion exists, the process advances to step S209;otherwise, the process advances to step S216.

If the process advances from step S208 to step S216, the errorregion/frequency determination unit 110 informs the selector 111 in stepS216 that the error detection unit 109 detects no error portion. Theselector 111 instructs the interpolation unit 112 to read out the MRimage of the frame of interest stored in the frame buffer 113. Theinterpolation unit 112 reads out the MR image of the frame of interestin accordance with the instruction, and outputs the readout MR image tothe selector 111. The selector 111 then outputs the MR image to thedisplay unit 115.

On the other hand, in step S209, the error region/frequencydetermination unit 110 determines whether to correct (it is possible tocorrect) the error portion detected in step S208.

Details of the determination process will be described here. The errorregion/frequency determination unit 110 refers to “the size of the errorportion” acquired during the process of detecting the error portion bythe error detection unit 109 and “a result determined by the errordetection unit 109 for an image of a past frame”. The errorregion/frequency determination unit 110 then determines whether “thesize of the error portion” is equal to or larger than a predeterminedthreshold. The error region/frequency determination unit 110 alsodetermines whether the error detection unit 109 determines that “theerror portion exists” in the MR image of a past frame (the immediatelypreceding frame in this embodiment) before the frame of interest. Thedetermination result for the past frame by the error detection unit 109is held by the error detection unit 109 itself.

If the determination result represents that “the size of the errorportion” is equal to or larger than the predetermined threshold and theerror detection unit 109 determines that “the error portion exists” inthe MR image of the frame immediately before the frame of interest, theerror region/frequency determination unit 110 determines that “it isimpossible to correct the error portion”.

If “the size of the error portion” is equal to or larger than thepredetermined threshold and the error detection unit 109 determines that“no error portion exists” in the MR image of the frame immediatelybefore the frame of interest (condition 1), the error region/frequencydetermination unit 110 determines that the MR image of the immediatelypreceding frame is substituted for that of the frame of interest.

If “the size of the error portion” is smaller than the predeterminedthreshold and the error detection unit 109 determines that “the errorportion exists” in the MR image of the frame immediately before theframe of interest (condition 2), the error region/frequencydetermination unit 110 determines that the error portion is corrected byusing the pixel group surrounding the error portion.

As described above, when an error portion exists within the MR image ofthe frame of interest, it can be determined whether it is possible tocorrect the error portion. If it is possible to correct the errorportion, how to correct it can be determined. Note that if “the size ofthe error portion” is smaller than the predetermined threshold and theerror detection unit 109 determines that “no error portion exists” inthe MR image of the frame immediately before the frame of interest, anymethod may be used to correct the error portion.

The above-described determination result by the error region/frequencydetermination unit 110 is output to the subsequent selector 111.

If the error region/frequency determination unit 110 informs theselector 111 that “it is impossible to correct the error portion”, theprocess advances to step S212 via step S210.

In step S212, the selector 111 instructs the OSD generation unit 114 tosuperimpose message information representing a warning on the physicalspace image which has been acquired by the OSD generation unit 114 fromthe image sensing unit 103 as the physical space image of the frame ofinterest. The selector 111 outputs to the display unit 115 the physicalspace image on which the message information is superimposed. In stepS216, the display unit 115 displays the physical space image on whichthe message information is superimposed.

If the above (condition 1) is satisfied, the process advances to stepS215 via step S210.

In step 215, the selector 111 instructs the interpolation unit 112 toread out the MR image of the frame immediately before the frame ofinterest stored in the frame buffer 113. The interpolation unit 112reads out the MR image in accordance with the instruction, and outputsthe readout MR image to the selector 111. The selector 111 outputs theMR image to the display unit 115. In step S216, the display unit 115displays, as the MR image of the frame of interest, the MR image of theframe immediately before the frame of interest.

If the above (condition 2) is satisfied, the process advances to stepS214 via step S210.

In step S214, the selector 111 instructs the interpolation unit 112 toread out the MR image of the frame of interest stored in the framebuffer 113. The interpolation unit 112 reads out the MR image inaccordance with the instruction, and corrects the error portion withinthe readout MR image by a process to be described later (a correctionprocess). The interpolation unit 112 outputs the corrected MR image tothe selector 111. The selector 111 outputs the corrected MR image to thedisplay unit 115. In step S216, the display unit 115 displays, as the MRimage of the frame of interest, the MR image within which the errorportion has been corrected by using the pixel group surrounding theerror portion.

A process in which the position and orientation information generationunit 106 generates position and orientation information of the imagesensing unit 103 from a physical space image will be described next.FIG. 3 is a view showing an example of a physical space sensed by theimage sensing unit 103. Referring to FIG. 3, reference numeral 303denotes a marker which includes two-dimensional barcodes obtained bycoding various information and a frame surrounding the barcodes. Themarker 303 is located on a table 302 as a physical object in FIG. 3.Reference numeral 301 denotes a physical space image acquired by sensingsuch physical space by the image sensing unit 103.

The image sensing unit 103 senses the physical space image 301, andoutputs the sensed physical space image 301 to the position andorientation information generation unit 106 via the wireless I/Fs 104and 105. The position and orientation information generation unit 106detects the marker 303 within the physical space image 301. The positionand orientation information generation unit 106 then analyzes thetwo-dimensional barcodes included in the detected marker 303, andrecognizes the identification information of the two-dimensionalbarcodes. On the image processing apparatus 102 side, thethree-dimensional coordinate position of the marker corresponding to theidentification information is stored in advance. When the identificationinformation is recognized with the process, the correspondingthree-dimensional coordinate position can be specified. When thecoordinate position of the marker 303 on the physical space image 301and the three-dimensional coordinate position are used, it is possibleto generate the position and orientation information of the imagesensing unit 103 by using a technique known in the field of, forexample, the photogrammetry. Note that a plurality of markers 303 may beused. An edge 304 within the image may be used in place of the marker303. Furthermore, the marker 303 is not limited to that including thetwo-dimensional barcodes and a color marker or a non-directional markersuch as a light-emitting element including an LED may be used. Asdescribed above, various objects and a combination thereof may be usedin place of the marker 303. The above-described process executed by theposition and orientation information generation unit 106 is a knowntechnique, and no more explanation will be given.

FIG. 4 is a view showing an example of an image transmitted/receivedbetween the video see-through HMD 101 and the image processing apparatus102.

Referring to FIG. 4, reference numeral 403 denotes a physical spaceimage sensed by the image sensing unit 103. This physical space image istransmitted to the image processing apparatus 102 side. The imageprocessing apparatus 102 superimposes a virtual space image on thephysical space image to generate an MR image 404, and transmits it tothe video see-through HMD 101. In this transmission, assume that anerror portion appears within the transmitted MR image, and that an errorMR image 405 is received on the video see-through HMD 101 side. As shownin the error MR image 405, the size of the error portion (the hatchedregion) within the error MR image 405 is large. It is difficult tocorrect the error portion using a pixel group surrounding it. Assumealso that an error portion exists within an image of a frame immediatelybefore the error MR image 405. In this case, the video see-through HMD101 superimposes message information representing a warning on thephysical space image 403 to generate an image 406 and displays it.

FIG. 5 is a view for explaining a process of substituting an MR image ofan immediately preceding frame for that of the frame of interest.Referring to FIG. 5, reference numeral 502 denotes an MR image of theframe of interest, which includes an error portion. If the size of theerror portion is equal to or larger than the threshold and no errorportion exists within an MR image 503 of a frame immediately before theframe of interest, the MR image 503 is displayed in place of the MRimage 502.

Such method is effective when there is no major change between the imagecontents of the frames, that is, when the amount of motion of the imagesensing unit 103 is small. In this case, a shift between an MR imagepresented to the right eye of the observer and that presented to theleft eye is small and a sense of stereoscopic viewing does notdeteriorate. It is therefore possible to provide a natural MR image forthe observer.

FIG. 6 is a view for explaining a process of correcting an error portionby using a pixel group surrounding it.

Referring to FIG. 6, assume that an error portion exists within a region602. Reference numeral 603 denotes a pixel as the error portion. Eightpixels 604 in the vicinity of the pixel 603 are used to correct thepixel 603. In this correction process, an average value of the pixelvalues of the eight pixels 604 is calculated and the average value isset as the pixel value of the pixel 603. In order to reduce the processload, an average value of the pixel values of two pixels adjacent toeach other in the vertical or horizontal direction may be set as thepixel value of the pixel 603.

Although FIG. 6 shows a case in which a pixel exists as the errorportion, the same process is executed when a plurality of neighboringpixels form the error portion. That is, an average value of the pixelvalues of pixels in the vicinity of the error portion is calculated. Theaverage value is weighted in accordance with the shortest distancebetween “a pixel forming the error portion” and “a pixel formingnon-error portion in the vicinity of the error portion” and the weightedaverage value is set as the pixel value of “the pixel forming the errorportion”.

As described above, there are various methods for correcting the pixelvalue of the pixel forming the error portion, and the present inventionis not limited to any of them.

[Second Embodiment]

In this embodiment, as an error portion correction method, a method ofcorrecting an error portion by substituting an MR image generated forthe other eye for an MR image of the frame of interest or a method ofcorrecting the error portion by using a pixel group surrounding it isused.

FIG. 7 is a block diagram showing a functional arrangement example of asystem according to this embodiment. As shown in FIG. 7, the systemaccording to this embodiment comprises an image processing apparatus 702and a video see-through HMD 701. The basic operation of the imageprocessing apparatus 702 is the same as that of an image processingapparatus 102 shown in FIG. 1, and the different points from the firstembodiment will be described below.

The video see-through HMD 701 will be described first. As shown in FIG.7, the video see-through HMD 701 includes a presentation unit forpresenting an image to the right eye of an observer wearing the videosee-through HMD 701 on the head, and a presentation unit for presentingan image to the left eye. Note that in the arrangement of the videosee-through HMD 701 shown in FIG. 7, R is appended to a reference numberof each component for presenting the image to the right eye of theobserver and L is appended to a reference number of each component forpresenting the image to the left eye. Assume that the components denotedby the same reference numbers without R or L perform the same operation.Although the components for presenting an image to one eye will bedescribed below, the same explanation can be applies to the componentsfor presenting an image to the other eye.

A 3D position sensor 704 measures the position and orientation ofitself. There are various kinds of sensors applicable to the 3D positionsensor 704, and the sensor is not limited to any of them. The positionand orientation information measured by the 3D position sensor 704 isoutput to a wireless I/F 705.

An image sensing unit 703R executes the same sensing operation as thatof the above-described image sensing unit 103. That is, the imagesensing unit 703R senses physical space images of frames, andsequentially outputs them to the subsequent wireless I/F 705.

The wireless I/F 705 wirelessly transmits, to the image processingapparatus 702, the position and orientation information received fromthe 3D position sensor 704 and the physical space image received fromthe image sensing unit 703R (an image sensing unit 703L).

On the image processing apparatus 702 side, a wireless I/F 706 outputs,to a subsequent CG rendering composition unit 708, the physical spaceimage and the position and orientation information which have beentransmitted from the video see-through HMD 701.

The CG rendering composition unit 708 calculates the position andorientation of the image sensing unit 703R by adding the position andorientation relationship between the 3D position sensor 704 and theimage sensing unit 703R, which has been measured in advance, to theposition and orientation represented by the position and orientationinformation received from the wireless I/F 706. The CG renderingcomposition unit 708 generates an image of a virtual space based on dataof contents 108, which is seen from the viewpoint having the calculatedposition and orientation. The CG rendering composition unit 708superimposes the generated image on the physical space image sensed bythe image sensing unit 703R and generates a right-eye MR image. Aleft-eye MR image is generated in the same manner. The right- andleft-eye MR images are output to the wireless I/F 706. The wireless I/F706 separately transmits these two MR images or transmits them as astream to the video see-through HMD 701.

Upon reception of the right- and left-eye MR images transmitted from theimage processing apparatus 702, the wireless I/F 705 outputs them toerror detection units 710R and 710L, respectively. The right- andleft-eye MR images are also output to an interpolation unit 713.

The error detection unit 710R executes the same process as in theabove-described error detection unit 109, and detects an error portionwithin the MR image.

An error region/frequency determination unit 711R refers to adetermination result by the error detection unit 710R. If the errordetection unit 710R detects no error portion, the error region/frequencydetermination unit 711R informs a selector 712R of it. If the errordetection unit 710R detects an error portion, by using “the size of theerror portion” acquired during the determination process and “a resultdetermined by the error detection unit 710R for an image of a pastframe”, the error region/frequency determination unit 711R executes adetermination process to be described later. This determination processis executed as follows. That is, it is determined whether the errorportion within the right-eye MR image of the current frame is to becorrected by substituting the left-eye MR image of the current frame forthe right-eye MR image of the same frame or by using a pixel groupsurrounding the error portion, or whether it is impossible to correctthe error portion. The error region/frequency determination unit 711Rinforms the subsequent selector 712R of the determination result.

If the error region/frequency determination unit 711R informs theselector 712R that the error detection unit 710R detects no errorportion, the selector 712R acquires “the right-eye MR image of thecurrent frame” received by the interpolation unit 713 from the wirelessI/F 705, and outputs the acquired image to a display unit 715R.

If the error region/frequency determination unit 711R informs theselector 712R that the left-eye MR image of the current frame is to besubstituted for the right-eye MR image of the same frame in order tocorrect the error portion, the selector 712R operates as follows. Thatis, the selector 712R acquires “the left-eye MR image of the currentframe” from the interpolation unit 713, and outputs the acquired imageto the display unit 715R.

If the error region/frequency determination unit 711R informs theselector 712R that the error portion is to be corrected by using thepixel group surrounding it, the selector 712R operates as follows. Thatis, the selector 712R acquires “the right-eye MR image of the currentframe” which has been received by the interpolation unit 713 from thewireless I/F 705, and corrects the error portion in the same way as inthe first embodiment. The selector 712R outputs the corrected MR imageto the display unit 715R.

If the error region/frequency determination unit 711R informs theselector 712R that it is impossible to correct the error portion, theselector 712R operates as follows. The selector 712R instructs an OSDgeneration unit 714R to superimpose message information representing awarning on the physical space image acquired by the OSD generation unit714R from the image sensing unit 703R as the physical space image of thecurrent frame. The selector 712R outputs to the display unit 715R thephysical space image on which the message information is superimposed.

Upon reception of the image from the selector 712R, the display unit715R displays the received image. The display unit 715R is attached tothe video see-through HMD 701 so as to be located in front of the eyes(the right eye) of the observer wearing the video see-through HMD 701 onthe head. The image output from the selector 712R is therefore displayedin front of the eyes (the right eye) of the observer wearing the videosee-through HMD 701 on the head.

FIG. 8 is a flowchart of a process executed by the video see-through HMD701. Note that the process according to the flowchart shown in FIG. 8 isthat of causing the display unit 715R to display an MR image of a frame,and in practice the process is repeated. This enables the display unit715R to display MR images of a plurality of frames. To cause the displayunit 715L to display an MR image of a frame, the same process isexecuted. Note that a process of causing the display unit 715R todisplay a right-eye MR image of the frame of interest (each of thesecond and subsequent frames) will be described below for descriptiveconvenience.

In step S801, the image sensing unit 703R senses a physical space seenfrom the position of the image sensing unit 703R itself in a sensingdirection to acquire an image of the frame of interest. The imagesensing unit 703R then outputs the sensed image to the wireless I/F 705and OSD generation unit 714R.

In step S802, the 3D position sensor 704 measures the position andorientation of itself, and outputs the measurement result as positionand orientation information to the wireless I/F 705.

In step S803, the wireless I/F 705 wirelessly transmits, to the imageprocessing apparatus 702, the image of the frame of interest acquired instep S801 and the position and orientation information acquired in stepS802.

The image processing apparatus 702 generates a right-eye MR image of theframe of interest by executing the above-described process, andwirelessly transmits the right-eye MR image to the video see-through HMD701 via the wireless I/F 706.

In step S809, the wireless I/F 705 receives the right-eye MR image ofthe frame of interest transmitted from the image processing apparatus702. The wireless I/F 705 outputs the received MR image to the errordetection unit 710R and interpolation unit 713.

In step S810, the error detection unit 710R determines whether an errorportion exists within the right-eye MR image of the frame of interest.The error detection unit 710R then informs the error region/frequencydetermination unit 711R of the determination result. If thedetermination result represents that the error portion exists, theprocess advances to step S811; otherwise, the process advances to stepS818.

If the process advances from step S810 to step S818, the errorregion/frequency determination unit 711R informs the selector 712R instep S818 that the error detection unit 710R detects no error portion.The selector 712R acquires “the right-eye MR image of the frame ofinterest” which has been received by the interpolation unit 713 from thewireless I/F 705, and outputs the acquired image to the display unit715R.

On the other hand, in step S811, the error region/frequencydetermination unit 711R determines whether the error portion detected instep S810 is corrected (it is possible to correct the error portion).

Details of this determination process will be described here. The errorregion/frequency determination unit 711R refers to “the size of theerror portion” acquired during the process of detecting the errorportion by the error detection unit 710R and “a result determined by theerror detection unit 710R for an image of a past frame”. The errorregion/frequency determination unit 711R determines whether “the size ofthe error portion” is equal to or larger than a predetermined threshold.The error region/frequency determination unit 711R also determineswhether the error detection unit 710R determines that “an error portionexists” in the MR image of the frame (the immediately preceding frame inthis embodiment) before the frame of interest. Assume that thedetermination result for the past frame by the error detection unit 710Ris held by the error detection unit 710R itself.

If the determination result represents that “the size of the errorportion” is equal to or larger than the predetermined threshold and theerror detection unit 710R determines that “the error portion exists” inthe MR image of the frame immediately before the frame of interest, theerror region/frequency determination unit 711R determines that “it isimpossible to correct the error portion”.

If it is determined that “the size of the error portion” is equal to orlarger than the predetermined threshold (condition 3), the errorregion/frequency determination unit 711R determines that the left-eye MRimage of the frame of interest is to be substituted for the right-eye MRimage of the frame of interest.

If it is determined that “the size of the error portion” is smaller thanthe predetermined threshold (condition 4), the error region/frequencydetermination unit 711R determines that the error portion is to becorrected by using the pixel group surrounding it.

As described above, when an error portion exists within an MR image ofthe frame of interest, it can be determined whether it is possible tocorrect the error portion. If it is possible to correct the errorportion, how to correct it can be determined.

The above-described determination result by the error region/frequencydetermination unit 711R is output to the subsequent selector 712R.

If the error region/frequency determination unit 711R informs theselector 712R that “it is impossible to correct the error portion”, theprocess advances to step S814 via step S812.

In step S814, the selector 712R instructs the OSD generation unit 714Rto superimpose message information representing a warning on thephysical space image which has been acquired by the OSD generation unit714R from the image sensing unit 703R as the physical space image of theframe of interest. The selector 712R outputs to the display unit 715Rthe physical space image on which the message information issuperimposed. In step S818, the display unit 715R displays the physicalspace image on which the message information is superimposed.

If the above (condition 3) is satisfied, the process advances to stepS817 via step S812.

In step 817, the selector 712R acquires “the left-eye MR image of theframe of interest” from the interpolation unit 713, and outputs theacquired image to the display unit 715R. In step S818, the display unit715R displays the left-eye MR image of the frame of interest as theright-eye MR image of the frame of interest.

If the above (condition 4) is satisfied, the process advances to stepS816 via step S812.

In step S816, the selector 712R acquires “the right-eye MR image of thecurrent frame” which has been received by the interpolation unit 713from the wireless I/F 705, and corrects the error portion in the sameway as in the first embodiment. The selector 712R then outputs thecorrected MR image to the display unit 715R. In step S818, the displayunit 715R displays, as the right-eye MR image of the frame of interest,the MR image within which the error portion has been corrected using thepixel group surrounding it.

FIG. 9 is a view for explaining a process of substituting an MR imagefor one eye for an MR image for the other eye in order to correct anerror portion within the MR image for the other eye. Referring to FIG.9, reference numeral 901L denotes left-eye MR images of frames; and901R, right-eye MR images of frames. Assume that an error portion isdetected within a left-eye MR image 902 and that the size of the errorportion is equal to or larger than a predetermined threshold. In thiscase, of the right-eye MR images 901R of the frames, a corresponding MRimage 903 of a frame is displayed in place of the left-eye MR image 902.Since the identical MR images are displayed for the right and left eyes,a sense of depth by stereoscopic images is not obtained. It is, however,unnecessary to buffer MR images of past frames unlike in the firstembodiment. Thus, the circuit arrangement is simple and low in price.

[Third Embodiment]

FIG. 10 is a block diagram showing a functional arrangement example of asystem according to this embodiment. As shown in FIG. 10, the systemaccording to this embodiment comprises an image processing apparatus1002 and a video see-through HMD 1001, which wirelessly communicate witheach other. The system is the same as that in the first or secondembodiment in this point.

The video see-through HMD 1001 will be described first.

A 3D position sensor 1005 is similar to the 3D position sensor 704 shownin FIG. 7. The 3D position sensor 1005 measures the position andorientation of itself, and outputs the measurement result as positionand orientation information to a subsequent position and orientationinformation generation unit 1006.

The position and orientation information generation unit 1006 calculatesthe position and orientation of an image sensing unit 1003 by adding theposition and orientation relationship between the 3D position sensor1005 and the image sensing unit 1003, which has been measured inadvance, to the position and orientation represented by the position andorientation information. The position and orientation informationgeneration unit 1006 then outputs position and orientation informationrepresenting the calculated position and orientation to a wireless I/F1007.

The image sensing unit 1003 is similar to the image sensing unit 103shown in FIG. 1, and senses physical space images of frames. The imagesensing unit 1003 sequentially outputs the sensed physical space imagesof the frames to a frame buffer 1004.

The wireless I/F 1007 wirelessly transmits, to the image processingapparatus 1002, the position and orientation information received fromthe position and orientation information generation unit 1006.

On the image processing apparatus 1002 side, upon reception of theposition and orientation information transmitted from the wireless I/F1007, a wireless I/F 1008 outputs the information to a subsequent CGrendering unit 1010.

The CG rendering unit 1010 generates an image (virtual space image) of avirtual space based on data of contents 108, which is seen from theviewpoint having the position and orientation represented by theposition and orientation information received from the wireless I/F1008, and outputs the generated virtual space image to the wireless I/F1008. The wireless I/F 1008 transmits the virtual space image to thevideo see-through HMD 1001.

The wireless I/F 1007 outputs, to an error detection unit 1012 and aninterpolation unit 1015, the virtual space image transmitted from theimage processing apparatus 1002.

Each time the virtual space image is received from the wireless I/F1007, the interpolation unit 1015 stores the received virtual spaceimage in a frame buffer 1016.

An error detection unit 1012 is similar to the error detection unit 109shown in FIG. 1, and checks (determines) whether an error portion (errorregion) exists within the virtual space image received from the wirelessI/F 1007. The error detection unit 1012 outputs the check result to asubsequent error region/frequency determination unit 1013.

The error region/frequency determination unit 1013 refers to thedetermination result by the error detection unit 1012. If the errordetection unit 1012 detects no error portion, the error region/frequencydetermination unit 1013 informs a selector 1014 of it. On the otherhand, if the error detection unit 1012 detects the error portion, byusing “the size of the error portion” acquired during the determinationprocess and “a result determined by the error detection unit 1012 for animage of a past frame”, the error region/frequency determination unit1013 executes a determination process. This determination process isexecuted as follows. That is, the error region/frequency determinationunit 1013 determines whether the error portion within the virtual spaceimage of the current frame is to be corrected by substituting a virtualspace image of the immediately preceding frame for the virtual spaceimage of the current frame or by using a pixel group surrounding theerror portion, or whether it is impossible to correct the error portion.The error region/frequency determination unit 1013 informs the selector1014 of the determination result.

If the error region/frequency determination unit 1013 informs theselector 1014 that the error detection unit 1012 detects no errorportion, the selector 1014 instructs the interpolation unit 1015 to readout the virtual space image of the current frame stored in a framebuffer 1016. The interpolation unit 1015 reads out the virtual spaceimage in accordance with the instruction, and outputs the readout imageto the selector 1014. The selector 1014 outputs the virtual space imageto a subsequent image composition unit 1017.

If the error region/frequency determination unit 1013 informs theselector 1014 that the virtual space image of the immediately precedingframe is to be substituted for that of the current frame in order tocorrect the error portion, the selector 1014 operates as follows. Theselector 1014 instructs the interpolation unit 1015 to read out thevirtual space image of the frame immediately before the current framestored in the frame buffer 1016. The interpolation unit 1015 reads outthe virtual space image in accordance with the instruction, and outputsthe readout image to the selector 1014. The selector 1014 outputs thevirtual space image to the subsequent image composition unit 1017.

If the error region/frequency determination unit 1013 informs theselector 1014 that the error portion is to be corrected by using thepixel group surrounding it, the selector 1014 operates as follows. Theselector 1014 instructs the interpolation unit 1015 to read out thevirtual space image of the current frame stored in the frame buffer1016. The interpolation unit 1015 reads out the virtual space image inaccordance with the instruction, and corrects the error portion withinthe readout virtual space image in the same way as the interpolationunit 112 of FIG. 1. The interpolation unit 1015 outputs the correctedvirtual space image to the selector 1014. The selector 1014 outputs thevirtual space image to the subsequent image composition unit 1017.

If the error region/frequency determination unit 1013 informs theselector 1014 that it is impossible to correct the error portion, theselector 1014 operates as follows. That is, the selector 1014 instructsand causes an OSD generation unit 1018 to superimpose messageinformation representing a warning on the physical space image which hasbeen acquired by the OSD generation unit 1018 from the frame buffer 1004as the physical space image of the current frame. The selector 1014outputs to the subsequent image composition unit 1017 the physical spaceimage on which the message information is superimposed.

If the image composition unit 1017 receives from the selector 1014 thephysical space image on which the message information is superimposed,it outputs the received image to a subsequent display unit 1019 withoutany change. On the other hand, if the image composition unit 1017receives the virtual space image from the selector 1014, it compositesthe virtual space image on the physical space image held in the framebuffer 1004 as the physical space image of the same frame as that of thevirtual space image, and generates an MR image. The image compositionunit 1017 outputs the MR image to the display unit 1019.

At this stage, the OSD generation unit 1018 holds the position andorientation of the image processing apparatus 1002 and range informationrepresenting an amount of shift from the position and orientation,within which the 3D position sensor 1005 can measure. Upon reception ofthe position and orientation information from the 3D position sensor1005, the position and orientation information generation unit 1006informs the OSD generation unit 1018 of the information. The OSDgeneration unit 1018 determines whether the difference obtained bysubtracting the position and orientation of the image processingapparatus 1002 from the position and orientation represented by theinformed position and orientation information falls within the rangerepresented by the range information. If the difference does not fallwithin the range, the OSD generation unit 1018 outputs, to the imagecomposition unit 1017, message information to prompt the observer tomove closer to the image processing apparatus 1002.

Upon reception of the message information, the image composition unit1017 superimposes the message information on the target image to beoutput to the display unit 1019.

The display unit 1019 is similar to a display unit 115 shown in FIG. 1.Upon reception of the image from the image composition unit 1017, thedisplay unit 1019 displays the received image.

FIG. 11 is a flowchart of a process executed by the video see-throughHMD 1001. The process according to the flowchart shown in FIG. 11 isthat of causing the display unit 1019 to display an MR image of a frame,and in practice the process is repeated. This enables the display unit1019 to display MR images of a plurality of frames. Note that a processof causing the display unit 1019 to display an MR image of the frame ofinterest (each of the second and subsequent frames) will be describedbelow for descriptive convenience.

In step S1101, the image sensing unit 1003 acquires an image of theframe of interest by sensing a physical space seen from the position ofthe image sensing unit 1003 in a sensing direction. The image sensingunit 1003 outputs the acquired image to the frame buffer 1004.

In step S1102, the 3D position sensor 1005 measures the position andorientation of itself, and outputs position and orientation informationas the measurement result to the position and orientation informationgeneration unit 1006.

In step S1103, the position and orientation information generation unit1006 generates position and orientation information representing theposition and orientation of the image sensing unit 1003 as describedabove.

In step S1104, the wireless I/F 1007 wirelessly transmits the positionand orientation information calculated in step S1103 to the imageprocessing apparatus 1002.

The image processing apparatus 1002 generates a virtual space image ofthe frame of interest using the position and orientation informationtransmitted in step S1104, and wirelessly transmits the generated imageto the video see-through HMD 1001 via the wireless I/F 1008.

In step S1105, the wireless I/F 1007 receives the virtual space image ofthe frame of interest transmitted from the image processing apparatus1002. The wireless I/F 1007 outputs the received virtual space image tothe error detection unit 1012 and interpolation unit 1015. Theinterpolation unit 1015 stores the virtual space image of the frame ofinterest in the frame buffer 1016.

In step S1106, the error detection unit 1012 determines whether an errorportion exists within the virtual space image of the frame of interest.The error detection unit 1012 informs the error region/frequencydetermination unit 1013 of the determination result. If thedetermination result represents that an error portion exits, the processadvances to step S1107; otherwise, the process advances to step S1112.

In step S1112, the error region/frequency determination unit 1013informs the selector 1014 that the error detection unit 1012 detects noerror portion. The selector 1014 instructs the interpolation unit 1015to read out the virtual space image of the frame of interest stored inthe frame buffer 1016. The interpolation unit 1015 reads out the virtualspace image of the frame of interest in accordance with the instruction,and outputs the readout image to the selector 1014. The selector 1014outputs the virtual space image to the image composition unit 1017. Theimage composition unit 1017 composites the virtual space image receivedfrom the selector 1014 on the physical space image held in the framebuffer 1004 as the physical space image of the same frame as that of thevirtual space image, and generates an MR image.

In step S1115, the OSD generation unit 1018 determines whether thedifference obtained by subtracting the position and orientation of theimage processing apparatus 1002 from the position and orientationrepresented by the position and orientation information informed fromthe position and orientation information generation unit 1006 fallswithin the range represented by the range information. If the differencefalls within the range, the process advances to step S1117; otherwise,the process advances to step S1116. In step S1116, the OSD generationunit 1018 outputs, to the image composition unit 1017, messageinformation to prompt the observer to move closer to the imageprocessing apparatus 1002.

In step S1117, the image composition unit 1017 outputs the MR imagegenerated in step S1112 to the display unit 1019. If, however, the imagecomposition unit 1017 receives “the message information to prompt theobserver to move closer to the image processing apparatus 1002”, theimage composition unit 1017 outputs to the display unit 1019 an MR imageon which the message information is superimposed.

In step S1107, the error region/frequency determination unit 1013determines in the same way as in the first embodiment whether the errorportion detected in step S1106 is to be corrected (it is possible tocorrect the error portion). The determination result by the errorregion/frequency determination unit 1013 is output to the subsequentselector 1014.

If the error region/frequency determination unit 1013 informs theselector 1014 that “it is impossible to correct the error portion”, theprocess advances to step S1109 via step S1108.

In step S1109, the selector 1014 instructs the OSD generation unit 1018to superimpose message information representing a warning on thephysical space image which has been acquired by the OSD generation unit1018 from the frame buffer 1004 as the physical space image of the frameof interest. The selector 1014 outputs to the image composition unit1017 the physical space image on which the message information issuperimposed.

If the above (condition 1) is satisfied, the process advances to stepS1110 via step S1108.

In step S1110, the selector 1014 instructs the interpolation unit 1015to read out the virtual space image of the frame immediately before theframe of interest stored in the frame buffer 1016. The interpolationunit 1015 reads out the virtual space image in accordance with theinstruction, and outputs the readout image to the selector 1014. Theselector 1014 outputs the virtual space image to the image compositionunit 1017.

In step S1113, the image composition unit 1017 composites the virtualspace image received from the selector 1014 on the physical space imageheld in the frame buffer 1004 as the physical space image of the sameframe as that of the virtual space image, and generates an MR image.

If the above (condition 2) is satisfied, the process advances to stepS1111 via step S1108.

In step S1111, the selector 1014 instructs the interpolation unit 1015to read out the virtual space image of the frame of interest stored inthe frame buffer 1016. The interpolation unit 1015 reads out the virtualspace image in accordance with the instruction, and corrects the errorportion within the virtual space image in the same way as in the firstembodiment. The interpolation unit 1015 outputs the corrected virtualspace image to the selector 1014. The selector 1014 outputs thecorrected virtual space image to the image composition unit 1017.

In step S1114, the image composition unit 1017 composites the virtualspace image received from the selector 1014 on the physical space imageheld in the frame buffer 1004 as the physical space image of the sameframe as that of the virtual space image, and generates an MR image.

In steps S1115 and S1116, the above-described processes are executed.

In step S1117, the image composition unit 1017 outputs, to the displayunit 1019, the MR image generated in step S1113 or S1114, or thephysical space image generated in step S1109. If, however, the imagecomposition unit 1017 receives “the message information to prompt theobserver to move closer to the image processing apparatus 1002”, theimage composition unit 1017 superimposes the message information on theimage, and outputs it to the display unit 1019.

[Fourth Embodiment]

In this embodiment, when an error is detected within a virtual spaceimage (CG image), the virtual space image is not rendered. In an MRapplication, a CG image which is not successfully rendered is anobstacle on a physical space image. Therefore, when an error is detectedwithin a CG image, the CG image is not rendered.

FIG. 13 is a block diagram showing a functional arrangement example of asystem according to this embodiment. As shown in FIG. 13, the systemaccording to this embodiment comprises a CG rendering apparatus 1303, animage processing apparatus 1302, and a video see-through HMD 1301.

The video see-through HMD 1301 will be described first.

An image sensing unit 1304 is similar to the image sensing unit 103described in the first embodiment, and senses a movie of a physicalspace. The sensed physical space image of each frame is output to theimage processing apparatus 1302.

A display unit 1305 displays the image output from the image processingapparatus 1302.

As is well known, a gyro sensor 1306 measures the position andorientation of itself. Any sensor other than the gyro sensor 1306 may beused. Position and orientation information as the measurement result bythe gyro sensor 1306 is output to the image processing apparatus 1302.

The image processing apparatus 1302 will be described next.

A position and orientation information generation unit 1307 is similarto the position and orientation information generation unit 106 shown inFIG. 1. The position and orientation information generation unit 1307uses the physical space image output from the image sensing unit 1304 togenerate position and orientation information representing the positionand orientation of the image sensing unit 1304, and outputs thegenerated position and orientation information to the CG renderingapparatus 1303.

The CG rendering apparatus 1303 generates a virtual space image usingthe position and orientation information. The generation process hasbeen described in the above embodiments, and a description thereof willbe omitted. The CG rendering apparatus 1303 outputs the generatedvirtual space image to the image processing apparatus 1302.

Upon reception of the virtual space image output from the CG renderingapparatus 1303, a CG/physical image composition unit 1308 composites thevirtual space image on the physical space image of the same frame asthat of the virtual space image. That is, the image processing apparatus1302 holds the physical space images, of several frames, output from thevideo see-through HMD 1301. Upon reception of the virtual space imagefrom the CG rendering apparatus 1303, the CG/physical image compositionunit 1308 acquires the physical space image of the same frame as that ofthe virtual space image from the held physical space images of theseveral frames. The CG/physical image composition unit 1308 thencomposites the acquired physical space image and the virtual space imagereceived from the CG rendering apparatus 1303. A technique forspecifying a corresponding frame is known, and a description thereofwill be omitted.

The virtual space image transmitted from the CG rendering apparatus 1303is also input to an abnormality determination unit 1309.

The abnormality determination unit 1309 determines whether an errorexists within the virtual space image received from the CG renderingapparatus 1303. The determination is made in various manners.

If, for example, the virtual space images, of several frames,sequentially output from the CG rendering apparatus 1303 are held andthere is no major change between the received virtual space images ofthe several frames, it is determined that an error exists. A techniquefor detecting a change between images is known. If, for example, anamount of motion of a specific object between images is smaller than acertain threshold, it is determined that there is no change between theimages.

On the CG rendering apparatus 1303 side, identification information toidentify an image of a frame, such as a time code, is embedded in avirtual space image. Upon reception of the virtual space image, theabnormality determination unit 1309 refers to the identificationinformation. If the pieces of referred identification information arenot continuous, it is determined that an error exists.

An amount of change between frames as a result of adding the positionand orientation relationship between the image sensing unit 1304 andgyro sensor 1306, which has been measured in advance, to a valuemeasured by the gyro sensor 1306, and an amount of change between theframes of the virtual space images received from the CG renderingapparatus 1303 are obtained. A difference Δ between the amounts ofchange is then calculated. If the calculated difference is equal to orlarger than a predetermined threshold, it can be determined that themotion of the video see-through HMD 1301 does not coincide with that ofthe virtual space image generated on the CG rendering apparatus 1303side. In this case, it is determined that an error exists.

A method of detecting whether an error exists is not limited to this, asa matter of course.

An image switch 1310 receives the determination result by theabnormality determination unit 1309. If an error exists, the imageswitch 1310 outputs the virtual space image received from the imagesensing unit 1304 to the display unit 1305; otherwise, the image switch1310 outputs the MR image generated by the CG/physical image compositionunit 1308 to the display unit 1305. In this way, the image switch 1310controls the display based on the determination result by theabnormality determination unit 1309. The display image may be graduallyfaded in/out instead of being suddenly switched.

The image switch 1310 may be omitted, and the composition ratio of aphysical space image to a virtual space image may be controlled inaccordance with the determination result by the abnormalitydetermination unit 1309. For example, as the absolute value of thedifference Δ is larger, composition with a higher ratio of the physicalspace image to the virtual space image is performed. As the absolutevalue of the difference Δ is smaller, the composition ratio of thephysical space image to the virtual space image is set closer to 1:1.

Referring to FIG. 13, although the image processing apparatus 1302 andthe CG rendering apparatus 1303 are separate apparatuses, they may beintegrated in a single apparatus. This applies to any of the aboveembodiments. A designer decides which function is given to eachapparatus, and also decides whether to use one or more apparatuses, asneeded. In any case, the system is practically the same.

FIG. 14 is a block diagram showing a functional arrangement example ofthe abnormality determination unit 1309.

An image FIFO 1311 can hold virtual space images of several past frames.

A comparator 1312 calculates a motion vector between images by using avirtual space image input as the latest frame and a virtual space imageof the immediately preceding frame held in the image FIFO 1311. Thecomparator 1312 outputs data of the calculated vector to a subsequentdiscriminator 1390.

A motion discriminator 1313 calculates a difference value between thepieces of position and orientation information sequentially output fromthe gyro sensor 1306, and outputs the difference value to thediscriminator 1390.

The discriminator 1390 compares the vector from the comparator 1312 withthe difference value from the motion discriminator 1313, and determineswhether the difference between them is equal to or larger than thethreshold. If the difference is equal to or larger than the threshold,the discriminator 1390 determines that an error exists, and outputs thedetermination result as an abnormality determination result.

[Fifth Embodiment]

In this embodiment, if an error is detected within a composite image ofa virtual space image and a physical space image, the physical spaceimage is displayed in place of the composite image.

FIG. 15 is a block diagram showing a functional arrangement example of asystem according to this embodiment. As shown in FIG. 15, the systemaccording to this embodiment comprises a video see-through HMD 1401 andan image processing apparatus 1402. Note that in FIG. 14, the samereference numerals as those in FIG. 13 denote the same components, and adescription thereof will be omitted.

The video see-through HMD 1401 will be described first.

A physical space image sensed by an image sensing unit 1304 is output tothe image processing apparatus 1402. A CG rendering composition unit1403 generates a virtual space image seen from the viewpoint having theposition and orientation represented by position and orientationinformation generated by a position and orientation informationgeneration unit 1307. The CG rendering composition unit 1403 compositesthe generated virtual space image on the physical space imagetransmitted from the video see-through HMD 1401, and generates an MRimage. The CG rendering composition unit 1403 then returns the generatedMR image to the video see-through HMD 1401.

An abnormality determination unit 1409 determines whether an errorexists within the MR image transmitted from the image processingapparatus 1402 in the same way as in the fourth embodiment.

The determination result is input to an image switch 1410. Along withthe MR image transmitted from the image processing apparatus 1402, thephysical space image from the image sensing unit 1304 is input to theimage switch 1410. The image switch 1410 refers to the determinationresult by the abnormality determination unit 1409. If an error exists,the image switch 1410 outputs the physical space image to a display unit1305; otherwise, the image switch 1410 outputs the MR image to thedisplay unit 1305.

FIG. 16 is a block diagram showing a functional arrangement example ofthe abnormality determination unit 1409.

An image FIFO 1411 holds the MR images, of several frames, output fromthe image processing apparatus 1402. A comparator 1412 compares the MRimage of the latest frame with that of the immediately preceding frameheld in the image FIFO 1411, and calculates the amount of shift betweenthe images. This process has been described in the fourth embodiment.

As a result of the comparison, if the shift amount is equal to or largerthan a threshold, the comparator 1412 determines that an error exists,and outputs the determination result.

The above-described embodiments may be used in combination.

[Other Embodiment]

The object of the present invention is achieved by the followingprocess. That is, a recording medium (storage medium) which recordssoftware program codes for implementing the functions of theabove-described embodiments is supplied to a system or apparatus. Thestorage medium is a computer-readable storage medium. A computer (or aCPU or MPU) of the system or apparatus reads out and executes theprogram codes stored in the recording medium. In this case, the programcodes read out from the recording medium implement the functions of theabove-described embodiments, and the recording medium which records theprogram codes constitutes the present invention.

The present invention includes a case in which the functions of theabove-described embodiments are implemented when the computer executesthe readout program codes and an operating system (OS) or the likerunning on the computer performs some or all of actual processes basedon the instructions of the program codes.

Furthermore, the present invention includes a case in which, after theprogram codes read out from the recording medium are written in thememory of a function expansion card inserted into the computer or thememory of a function expansion unit connected to the computer, the CPUof the function expansion card or function expansion unit performs someor all of actual processes based on the instructions of the programcodes and thereby implements the functions of the above-describedembodiments.

When the present invention is applied to the recording medium, therecording medium stores the program codes corresponding to theabove-described flowcharts.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2007-142324 filed May 29, 2007, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An apparatus comprising one or more processorscarrying out functions of a first obtaining unit, a second obtainingunit, a generation unit, a synthesizing unit, a determination unit, anda display control unit; the first obtaining unit obtaining a physicalspace image obtained by sensing a physical space by an image sensingdevice; the second obtaining unit obtaining a position and orientationof the image sensing device; the generation unit generating a virtualobject based on the position and orientation obtained by the secondobtaining unit; the synthesizing unit generating a synthesized image bysynthesizing the physical space image with the virtual object; thedetermination unit determining whether an error region exists in thesynthesized image; and the display control unit causing a display deviceto display the synthesized image in a case where the determination unitdoes not determine that the error region exists in the synthesizedimage, and to cause the display device to display the physical spaceimage in a case where the determination unit does determine that theerror region exists in the synthesized image.
 2. The apparatus accordingto claim 1, wherein the image sensing device is attached on a headmounted display.
 3. The apparatus according to claim 1, wherein thedisplay device is attached on a head mounted display.
 4. The apparatusaccording to claim 1, further comprising: a storage configured to storethe synthesized image; wherein the determination unit is furtherconfigured to determine whether an error region exists in a synthesizedimage of interest by comparing the synthesized image of interest and asynthesized image which was previously synthesized and stored in thestorage.
 5. The apparatus according to claim 4, wherein thedetermination unit is further configured to determine whether the errorregion exists in the synthesized image of interest by comparing thesynthesized image of interest and a synthesized image which wassynthesized immediately before the synthesized image of interest issynthesized.
 6. A head mounted display which connects with an imageprocessing apparatus, comprising: an image capturing device whichcaptures a physical space image; a transmitter configured to transmitthe physical space image to the image processing apparatus; one or moreprocessors carrying out functions of an obtaining unit, a determinationunit, and a display control unit; the obtaining unit obtaining asynthesized image generated by synthesizing the physical space imagewith a virtual object from the image processing apparatus via thetransmitting unit; and the determination unit determining whether anerror region exists in the synthesized image; and a display control unitcausing a display device to display the synthesized image in a casewhere the determination unit does not determine that the error regionexists in the synthesized image, and causing the display device todisplay the physical space image in a case where the determination unitdoes determine that the error region exists in the synthesized image. 7.A system comprising: a display apparatus ; an image processing apparatuswhich connects with the display apparatus and generates a virtual imageto be displayed in the display apparatus; wherein the display apparatusfurther comprises; an image capturing device which captures a physicalspace image; a first transmitter configured to transmit the physicalspace image to the image processing apparatus; wherein the imageprocessing apparatus further comprises; a first receiver configured toreceive the physical space image; one or more first processors carryingout functions of an obtaining unit, a generation unit, a synthesizingunit; the obtaining unit obtaining a position and orientation of theimage display apparatus; the generation unit generating a virtual objectbased on the position and orientation; the synthesizing unit generatinga synthesized image by synthesizing the physical space image with thevirtual object; and a second transmitter configured to transmit thesynthesized image to the display apparatus; wherein the displayapparatus further comprises; a second receiver configured to receive thesynthesized image from the image processing apparatus; one or moresecond processors carrying out functions of a determination unit and adisplay control unit; the determination unit determining whether anerror region exists in the synthesized image; and the display controlunit causing a display device to display the synthesized image in a casewhere the determination unit does not determine that the error regionexists in the synthesized image, and causing the display device todisplay the physical space image in a case where the determination unitdoes determine that the error region exists in the synthesized image. 8.The apparatus according to claim 7, wherein the display apparatus is ahead mounted display.
 9. A method comprising: obtaining a physical spaceimage by sensing a physical space by an image sensing device; obtaininga position and orientation of the image sensing device; generating avirtual object based on the obtained position and orientation;generating a synthesized image by synthesizing the physical space imagewith the virtual object; determining whether an error region exists inthe synthesized image; and causing a display device to display thesynthesized image in a case where it is not determined that the errorregion exists in the synthesized image, and causing the display deviceto display the physical space image in a case where it is determinedthat the error region exists in the synthesized image.
 10. Anon-transitory computer-readable medium storing a program for causing acomputer to: obtain a physical space image by sensing a physical spaceby an image sensing device; obtain a position and orientation of theimage sensing device; generate a virtual object based on the obtainedposition and orientation; generate a synthesized image by synthesizingthe physical space image with the virtual object; determine whether anerror region exists in the synthesized image; and cause a display deviceto display the synthesized image in a case where it is not determinedthat the error region exists in the synthesized image, and cause thedisplay device to display the physical space image in a case where it isdetermined that the error region exists in the synthesized image.
 11. Amethod comprising: capturing a physical space image; transmitting thephysical space image to an image processing apparatus; obtaining asynthesized image from the image processing apparatus, wherein thesynthesized image is obtained by synthesizing the physical space imagewith a virtual object; determining whether an error region exists in thesynthesized image; and causing a display device to display thesynthesized image in a case where it is not determined that the errorregion exists in the synthesized image, and causing the display deviceto display the physical space image in a case where it is determinedthat the error region exists in the synthesized image.
 12. Anon-transitory computer-readable medium storing a program for causing acomputer to: capture a physical space image; transmit the physical spaceimage to an image processing apparatus; obtain a synthesized image fromthe image processing apparatus, wherein the synthesized image isobtained by synthesizing the physical space image with a virtual object;determine whether an error region exists in the synthesized image; andcause a display device to display the synthesized image in a case whereit is not determined that the error region exists in the synthesizedimage, and cause the display device to display the physical space imagein a case where it is determined that the error region exists in thesynthesized image.
 13. A method comprising: capturing a physical spaceimage at a display apparatus; transmitting the physical space image fromthe display apparatus to an image processing apparatus; obtaining aposition and orientation of the image display apparatus; generating avirtual object based on the position and orientation; generating asynthesized image by synthesizing the physical space image with thevirtual object; transmitting the synthesized image from the imageprocessing apparatus to the display apparatus; determining whether anerror region exists in the synthesized image; and causing a displaydevice to display the synthesized image in a case where it is notdetermined that the error region exists in the synthesized image, andcausing the display device to display the physical space image in a casewhere it is determined that the error region exists in the synthesizedimage.
 14. A non-transitory computer-readable medium storing a programfor causing a computer to: capture a physical space image at a displayapparatus; transmit the physical space image from the display apparatusto an image processing apparatus; obtain a position and orientation ofthe image display apparatus; generate a virtual object based on theposition and orientation; generate a synthesized image by synthesizingthe physical space image with the virtual object; transmit thesynthesized image from the image processing apparatus to the displayapparatus; determine whether an error region exists in the synthesizedimage; and cause a display device to display the synthesized image in acase where it is not determined that the error region exists in thesynthesized image, and cause the display device to display the physicalspace image in a case where it is determined that the error regionexists in the synthesized image.
 15. A head mounted display whichconnects with an image processing apparatus, comprising: an imagecapturing device which captures a physical space image; a transmitterconfigured to transmit the physical space image to the image processingapparatus; one or more processors carrying out functions of an obtainingunit and a display control unit; the obtaining unit obtaining asynthesized image generated by synthesizing the physical space imagewith a virtual object from the image processing apparatus via thetransmitting unit; and a display control unit causing a display deviceto display the synthesized image in a case where an error does not existwith the synthesized image, and causing the display device to displaythe physical space image in a case where an error exists with thesynthesized image.
 16. The head mounted display according to claim 15,wherein the one or more processors further carry out a function of adetermination unit, the determination unit determining whether an errorexists with the synthesized image.
 17. The head mounted displayaccording to claim 15, wherein the determination unit determines thatthe error exists with the synthesized image in a case where there is nochange in a predetermined number of frames of the synthesized imageobtained by the obtaining unit.
 18. The head mounted display accordingto claim 16, wherein the determination unit determines that the errorexists with the synthesized image in a case where a shift amount betweenthe synthesized image obtained by the obtaining unit and an immediatelypreceding frame of the synthesized image is not less than a threshold.19. A system comprising: a display apparatus; an image processingapparatus which connects with the display apparatus and generates avirtual image to be displayed in the display apparatus; wherein thedisplay apparatus further comprises; an image capturing device whichcaptures a physical space image; a first transmitter configured totransmit the physical space image to the image processing apparatus;wherein the image processing apparatus further comprises; a firstreceiver configured to receive the physical space image; one or morefirst processors carrying out functions of an obtaining unit, ageneration unit, a synthesizing unit; the obtaining unit obtaining aposition and orientation of the image display apparatus; the generationunit generating a virtual object based on the position and orientation;the synthesizing unit generating a synthesized image by synthesizing thephysical space image with the virtual object; and a second transmitterconfigured to transmit the synthesized image to the display apparatus;wherein the display apparatus further comprises; a second receiverconfigured to receive the synthesized image from the image processingapparatus; one or more second processors carrying out a function of adisplay control unit; and the display control unit causing a displaydevice to display the synthesized image in a case where an error doesnot exist with the synthesized image, and causing the display device todisplay the physical space image in a case where an error exists withthe synthesized image.
 20. A method comprising: capturing a physicalspace image; transmitting the physical space image to an imageprocessing apparatus; obtaining a synthesized image from the imageprocessing apparatus, wherein the synthesized image is obtained bysynthesizing the physical space image with a virtual object; and causinga display device to display the synthesized image in a case where anerror does not exist with the synthesized image, and causing the displaydevice to display the physical space image in a case where an errorexists with the synthesized image.
 21. A non-transitorycomputer-readable medium storing a program for causing a computer to:capture a physical space image; transmit the physical space image to animage processing apparatus; obtain a synthesized image from the imageprocessing apparatus, wherein the synthesized image is obtained bysynthesizing the physical space image with a virtual object; and cause adisplay device to display the synthesized image in a case where an errordoes not exist with in the synthesized image, and cause the displaydevice to display the physical space image in a case where an errorexists with the synthesized image.